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Abstract 

This paper describes our work exploring the suit- 
ability of formal specification mctho<ls for indepen- 
dent verification and validation (IV&V) of software 
specifications for large . safety critical systems. An 
IV&V contractor often has to perform rapul analy- 
sis on incomplete specifications . with no control over 
how those specifications arc represented . Lightweight 
formal methods show significant promise in this con- 
text, as they offer a t pay of uncovering major errors, 
without the burden of full proofs of correc tness . We de- 
scribe an experiment in the application of the method 
SCR to testing for consistency properties of a partial 
model of the requirements for Fault Detection Isola- 
tion and Recovery on the space station. We conclude 
that the insights gained from formalizing a specifica- 
tion is valuable . and it is the process of formalization, 
rather than the end product that is important. It was 
only necessary to build enough of the formal motlcl to 
test the properties in which we were, interested. Main- 
tenance of fidelity between multiple representations of 
the same requirements (as they evolve) is still a prob- 
lem. and deserves further study. 

1 Introduction 

Requirements engineering methods typically pro- 
vide a set of notations for expressing software specifi- 
cations. together with tools for chocking properties of 
specifications, such as completeness and consistency. 
Ifi general, such methods demand a full commitment. 
It is assumed that the method will be used to con- 
struct a complete specification, which will then act as 
a baseline for subsequent development phases. How- 
ever. to validate and verify large specifications for 
safety-critical real-time systems, it is sensible to ap- 
ply a number of different methods, to overcome weak- 
nesses and biases of each individual method. For ex- 
ample, a formal method might be used to model a 
critical portion of an informal specification, to chock 
safety and livcncss properties of that portion. In order 
to manage the application of multiple methods, it is 
necessary to develop and maintain alternative repre- 
sentations of partial specifications, and to express the 
relations! lips between them. 

This paper describes some preliminary work on the 
use of formal specification as a tool for Independent 


Verification and Validation (IV&V). Our intention is 
to use formal methods not as a part of the develop- 
ment process itself, but as a ‘shadow activity, per- 
formed by an independent team of experts. Our long- 
term expectation is that this approach will turn out 
to he a less painful way of introducing formal methods 
into well-established, large-scale software development 
processes. 

There are a number of questions that need to bo 
addressed before formal methods can be used in this 
way. Most published ease studies of formal methods 
have focussed on the use of a formal specification as 
a baseline from wliich design and code can be verified 

a In contrast, wc have been applying formal mctli- 
s for intermittent "spot chocks’* to test for errors as 
the requirements evolve. The term “lightweight formal 
methods" has boon used to describe tliis approach [15]. 
In this context, the the formal specification is dispens- 
able - what is important arc the insights gained from 
the process of formalizing partial views of the require- 
ments and from validating properties of the resulting 
models. However, it is still necessary to demonstrate 
fidelity between the original (informal) specification, 
and the formal model. Furthermore, iterative applica- 
tion of this approach can be greatly facilitated if the 
relationships between the partial views are captured. 

The context for tliis work is the development of soft- 
ware for the International Space Station (ISS) project. 
Docing Space and Defense Group Houston (Prime) 
is responsible for supervising the overall development 
and integration of International Space Station soft- 
ware. There arc three Product Groups (PGs). McDon- 
nell Douglas Aerospace. Rockwell Aerospace - Rockct- 
dync and Boeing Space and Defense Group Huntsville, 
who arc developing several key Computer Software 
Configuration Items (CSCIs). There arc also several 
International Partners (IPs) including Russia. .Jfipan, 
Canada, and the European Space Agency, who arc 
developing software that will need to be incorporated 
into ISS. With over 45 flight computers and an esti- 
mated 1.1 million source lines of flight code, the po- 
tential problems arc considerable. Software IV&V is 
currently being performed by Intcrrnctrics. under an 
interim contract. The Intcrmctrics team is based at 
Fairmont. W.Va.. with personnel stationed in Houston 
and Huntsville in order to interact with the develop- 



mcnt teams. 

In section 2, we outline the IV&V process, and dis- 
cuss the aspects of tliis process that hinder effective 
IV&V. With tliis as background, the remainder of the 
paper focuses on the use of methods and tools within 
this process. We present two experiments in the use 
of formal specification. For these we used a combi- 
nation of AN D/O R tables [8h and the Software Cost 
Reduction (SCR) approach [9]. The first experiment 
involved the translation of a portion of the Fault De- 
tection, Isolation and Recovery (FDIR) specification 
into a formal notation. This experiment confirmed 
that the natural language used in the Software Re- 
quirements Specification (SRS) documents is inher- 
ently ambiguous, and that the task of generating for- 
mal specifications from this documentation is fraught 
with difficulty. In the second experiment, we applied 
an automated consistency checking tool, to test some 
formal properties of the specification. Although this 
experiment demonstrated that important disjointness 
properties did not hold, the results did not add any 
more value to the analysis. The first experiment had 
already demonstrated that the way in winch these re- 
quirements were expressed was a problem. The errors 
found in the second experiment were attributable to 
the same problem. 

Application of formal methods in this context was 
not always easy. The informal specification from 
which we derived our models did not permit an easy 
translation into a state-based model. We encountered 
severe problems in demonstrating fidelity, and provid- 
ing traceability between the two. Section 5 <liscusscs 
these problems, and sketches out further work aimed 
at eliciting relationships between partial specifications 
by extracting information from fine-grained process 
capture. 

We conclude that in an IV&V context, the ana- 
lytical benefits offered by formal methods have to be 
weighed against the effort needed to maintain fidelity 
between a formal model and the informal specifica- 
tion used by the development team. An IV&V team 
ncc<ls to be able to perform partial analyses on partial 
specifications, without being tied to any one formal- 
ism. The analysis carried out must be sufficient to 
reveal important problems, as opposed to surface de- 
fects. Further analysis is a waste of effort until these 
problems have been fixed. Tliis conclusion implies a 
change of perspective for the use of formal methods: 
while the specification is still evolving it is important 
to identify quickly any major defects; it is not neces- 
sary to perform a complete analysis. Tools that arc 
geared towards finding and char act cri zing such proli- 
lems (E.g. SCR* [10], Nitpick [11]. etc.) arc more 
useful than tools geared towards proving correctness 
(E.g. theorem provers). 

2 The IV&V Process 

For Independent Verification and 

Validation (IV&V). the software customer hires a sep- 
arate contractor to analyze the products and process 
of the software development contractor. Tliis analysis 
is performed in parallel with the development process, 
throughout the software lifecycle, and in no way re- 


places in-house V&V. IV&V is applied in high-cost 
and safety-critical projects to overcome analysis bias 
and reduce development risk. The customer relies on 
the IV&V contractor as an informed, unbiased advo- 
cate to assess the status of a project s schedule, cost, 
and the viability of its product during development. 
In full IV&V. the IVfeV contractor has managerial, 
financial and technical independence, and reports to 
the customer, not the developer. Most importantly, 
the IV&V contractor should be engaged as cjirly as 
possible in the project: studies have shown that IV&V 
has the biggest impact in the early phases, especially 
in the requirements phase [13]. 

An example IV&V activity is the analysis of spec- 
ifications on the Space Station project. An SRS is 
written by the relevant development contractor for 
each Software Configuration Item (CSCI). These are 
written in natural language, and follow the format 
of DOD-STD-2167A. The IV&V contractor periodi- 
cally receives copies of the SRS documents, in various 
stages of completion. These arc analyzed for tcdinical 
integrity by the IV&V contractor, in order to iden- 
tify any requirements problems and risks. The kind of 
analysis performed will vary according to the level and 
the type of specification, and will cover issues such as 
clarity, testability, traceability, consistency and com- 
pleteness. If problems are identified, the IV&V con- 
tractor may recommend that either the requirements 
be rewritten, or the problem be tracked through sub- 
sequent phases. 

Performing IV&V on large projects is far from 
straightforward. Problems faced by the IV&V con- 
tractor include: 

resource allocation - A complete, detailed analysis 
of the entire system is infeasible. Effort has to 
he allocated so as to maximize effectiveness. For 
example, a criticality and risk analysis might be 
performed to determine which components need 
the most scrutiny. Timing is also a factor; effort 
needs to be allocated at the right points in the 
development of a product (e.g. a document), so 
that the product is mature enough to be analyzed, 
but not so mature that it cannot be changed. 

short timescales - To be most effective. IV&V re- 
ports arc needed as quickly as possible. There is 
always a delay between the delivery of an interim 
product to the TV&V team, and the completion 
of analysis of that product. During this time, the 
development process continues. Hence, if IV&V 
analysis takes too long, the results might be avail- 
able too late to be useful. In general, the earlier 
an error is reported, the cheaper it is to correct. 

lack of access - Contact between the development 
team and the IV&V team is difficult to manage. 
The IV&V team needs to maintain independence, 
whilst ensuring they obtain enough information 
from the developers to do their job. PYom the 
developed point of view, interaction with the 
IV&V team represents a cost overhead, which can 
interfere with project dea<llincs. Inevitably, the 
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IV&V contractor has less access to the develop- 
ment team than is ideal. 

evolving products - Documentation from the de- 
velopment. team is usually made available to the 
IV&V contractor in draft form, to facilitate early 
analysis. The drawback is that documents may be 
revised while the IV&V team is analyzing them, 
making the results of the analysis irrelevant be- 
fore it is finished. 

reporting the right problems - The IV&V con- 
tractor has. by necessity, considerable discretion 
over the kinds of analysis to perform on different 
products. It also has discretion over which prol>- 
Icms to report. It is vital to the effective use of 
IV&V that, the IV&V contractor prioritizes the 
problems it identifies. If too many trivial prol>- 
Icms are reported, this may swamp the communi- 
cation channels with the developer and the cus- 
tomer. 

lack of voice - The IV&V contractor may have 
difficulty in getting its message across, espe- 
cially when the development contractor disputes 
IV&V's assessment. Often, problems found by 
IV&V have cost and schedule implications, and 
in such circumstances the customer may be more 
willing to listen to assurances from the developer. 
The effectiveness of IV&V then depends on hav- 
ing a high-placed advocate within the customer 
organization. 

Despite these problems. IV&V has been shown to 
he a cost-effective means of improving the quality of 
the software product, and providing extra assurance 
for high-cost, safety -critical projects [12]. In addition 
to providing analysis of project artifacts (c.g. require- 
ments. code, test plans), the presence of IV&V in the 
lifecycle also has a positive effect on the quality of 
the software. Our work suggests that the inferorhon 
between the IV&V and development teams drives im- 
provements in both products and processes. This ef- 
fect. however, is difficult to capture and quantify. 

3 Methods and Tools in IV&V 

An important aspect of IV&V is the choice of the 
right methods and tools. Ideally, an IV&V contractor 
will have access to all the tools used by the develop- 
ment team, including the ability to share all project 
datal>ascs. However, the IV&V team also needs to 
supplement these with additional methods and tools, 
to address any gaps or weaknesses in the coverage of 
the developer's tools. These additional tools need to 
complement the developers tools, so that interoper- 
ability docs not become a problem. The use of these 
additional tools is an important factor in ensuring that 
IV&V is truly independent. 

It is often the case that the use of a particular 
method or tool by the IV&V team leads to the adop- 
tion of that method or tool hy the developers. In part 
this is due to the ‘watchdog effect*: if the developers 
know that their product will he analyzed in a partic- 
ular way. it is in their interest to perform the analysis 


themselves before releasing it. If this seems to he a 
rather negative reason to adopt a technique, there is 
also a positive aspect. Because the IV&V team is out 
of the critical path for the software development ef- 
fort. they have more scope for experimentation with 
new techniques than the developers [1]. Hence, in 
some ways the IV&V team can play a role as a prov- 
ing ground for new techniques, and can come to he 
an agent of process improvement. For these reasons, 
we believe that IV&V offers a practical route through 
which formal methods may he introduced into projects 
that would otherwise not he able to adopt them. 

There are still problems to be overcome whenever 
the IV&V team adopts a tool that is not used hy the 
developers. Compatibility with the developers’ tools 
is important. For example, if the IV&V temn uses 
a formal specification tool, the informal specification 
delivered hy the developers will need to he translated 
into the formal specification language not just once, 
hut each time the developers produce a new draft. 
Any problems identified hy using the tool must he 
traced back to the informal specification, before they 
can he reported. There must he a reasonable assur- 
ance that the formal specification remains faitliful to 
the original, otherwise any analysis performed on it 
is wortlilcss. Hence, keeping track of the relationship 
between the formal and informal specifications is vital. 

4 Experiments with formal methods 

Having described the role that an IV&V contractor 
plays in the software process, and outlined the issues 
involved in the selection of tools and techniques for 
IV&V. we now present our work on the use of fonnal 
methods in the IV&V of requirements specifications. 
We performed two experiments. The first was a for- 
maliscition of individual requirements statements into 
a tabular form, to improve clarity. The second was the 
development of a formal model of these requirements, 
which was then tested for consistency. 

Currently, the development contractors on the 
Space Station project use natural language specifica- 
tions extensively. We are working with the IV&V team 
to explore how formal methods can enhance the kinds 
of analysis they perform on the developer’s informal 
specifications. Here, we will report our work with the 
Fault Detection. Isolation and Recovery requirements 
for the main command and control bus. An example 
requirement is given in figure 1. 

Our initial interest in formal methods was twofold. 
First, it was clear that the informal specifications were 
hard to understand, and would benefit from a clearer 
representation. We needed a notation that was both 
precise and easy to read. Lcvcson s AND/OR tables 
18] provided us with a solution. During the develop- 
ment of the RSML specifications for TCAS n. Lcvcson 
adopted these AND/OR tables in preference to predi- 
cate calculus, as they were readable by a wide rmige of 
people. Tliis tabular representation was well suited to 
the Space Station FDIR requirements (see table 1), as 
it mapped directly onto the individual requirements 
statements. 

Second, we needed a way to verify that the specified 
functionality was internally consistent. For the FDHl 
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(2.16.3.f) While acting a* the him controller, the C&C MDM CSCI shall aet the e.c.w. 
indicator identified in Table 3.2.16-11 for the corresponding RT to -failed' and aet the 
failure atatua to “failed' for all RT’a on the bua upon detection of tranaaction error* of 
selected me**age* to RT* who*e 1553 FD1R i« not inhibited in two consecutive processing 
frame* within 100 miUi*ec of detection of the second transaction error if; a backup DC is 
available, the DC ha* been switched in the last 20 sec. the SPD card reaet capability is 
inhibited, or the SPD card ha* been reset in the last 10 major (10-«ernnd) frames, and 
either: 

1. the transaction error* are from multiple RT’*. the current channel has been reset 
within the lafit major frame, or 

2. the transaction error* are from multiple RT’», the bus channel'* reset capability is 
inhibited, and the current channel ha* not been reset within the last major frame. 


p; C urc 1- An example of a level 3 requirement, for FDIR of the Command andControl bus for Space Station This 
requirement specifies the circumstance* under wliich all remote terminals (RTs) on the bus should be switched 

to their backups. _ __ — — 


requirements, this meant checking that the conditions 
specified for each recovery action were mutually ex- 
clusive. and that the requirements covered all possi- 
ble conditions. Hand checking these properties would 
have been hard, so we sought a tool to help We ex- 

amined several tools, before selecting SCR lluj. bClv 

offered two important advantages. First, the nota- 
tion was primarily tabular, which appeared to be an 
important aid to readability. Second, tlic tool bad au- 
tomated checking for properties sucli as coverage anil 
disjointess of a state based model [9]. In addition, 
this tool did not require us to build a complete formal 
model of the Bus FDIR functionality in order to check 
these properties. 

4*1 Experiment 1: Translation 

Our first experiment concerned the translation of 
requirements like that shown in Figure 1 into a formal 
notation. Lcvcson’s AND/OR tables allowed us to 
represent arbitrary combinations of conjunctions and 
disjunctions without ambiguity, and in a form that 
was clearly readable. Table 1 shows the tabular form 
of the requirement in Figure 1. 

For the IV&V team, this was a significant improve- 
ment. in readability. More importantly, the process of 
producing the tables ensured that the analysts fully 
understood the requirement. Tliis benefit is very im- 
portant for IV&V. In many cases, just reading a spec- 
ification is insufficient to really appreciate the de- 
tail Short of repeating the development process from 
scratch, it can he hard for the IV&V analyst to under- 
stand a specification in the same way that its authors 
understand it. Translating it into a table, however, 
proved to be a valuable clarification process. 

There was. unfortunately, a problem. Translation 
of a single requirement, like the one above, was not a 
straightforward task. Translation of tliis requirement 
took several attempts until wc were happy with the 
table, and oven then wc were not convinced that it 
was right. 

Wc conducted an experiment to investigate the 
problem. We gave the English language version to 


four different people, all of whom had some experience 
of representing requirements using tables, and asked 
them to produce the tabular form. Two of these peo- 
ple were domain experts, and two were not. Wc were 
interested in exploring the scope for misinterpretation 
of the requirements from the point of view of both do- 
main experts who write such requirements, and other 
stakeholders, such as the programmer who would have 
to implement them. 

We received four different answers. These differed 
in both the number of conditions identified (i.c. num- 
ber of rows in the table) and the number of combina- 
tions under which the function would be activated (i.c. 
columns in the table). The version shown in Table 1 
is a synthesis of the four answers, representing what 
wc currently believe is the intended interpretation. 

The differences in the responses show that the origi- 
nal requirement is riddled with ambiguities. For exam- 
ple. the mixture of ‘ands* and ‘ors’ in the requirement 
is a problem because, unlike programming languages. 
English docs not have any standard precedence rules. 
It is not clear how to scope the various subclnuscs. 
cither. For example, the timing condition ‘within 100 
milliscc...’ could refer to the inhibition of the FDIR.. 
or to one or both of the required setting operations. 
With a little domain knowledge, it is possible to elim- 
inate some interpretations, hut tliis is by no means a 
trivial task, and there is no guarantee that everyone 
who needs to read this requirement will get it right. 

The experiment demonstrated tlirre important re- 
sults. Firstly, the tabular forms were very helpful in 
resolving misunderstandings. For example, it would 
he difficult to discover that our four subjects had dif- 
ferent interpretations of the original requirement with- 
out asking them to rc-writc it. By re-writing it in 
tabular form, we could identify exactly where the dis- 
agreements lay. and then take each discrepancy in turn 
and discuss what we thought the most likely inter- 
pretation was. FVom this, we were able to synthesize 
a ‘host* interpretation. Obtaining individual transla- 
tions and comparing them was more effective in identi- 
fying differences in our understandings than our initial 
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C&C MUM acting as the bus controller 


"T 

m ■ 

Detection of transaction errors 


T 

in two consecutive processing frames 


m 1 

errors are on selected messages 


I 

m— 

the RTs 1053 KUill is not inhibited 


T 

A backup tJU is available m , 



Thr TTTTFIas been switched in the last zu seconds 


m ■ 

The SFU card reset capability is inhibited ^ 


I 

The SPL) card lias been reset m the last 1U major 


* 

(10 second) frames 



The transaction errors arc from multiple 1C1 s _ 



The current, channel has been reset within the last 


1 

major frame . . 




'The bus channel s reset capability is inhibited 









mil * LovcBon-stvlc table for requirement 2.16.3.f. Tins table summarizes the romlitional part of the 
? .howmg tour ?oml,in»tioi» of con.li.ion. (tl.c four column.) on, lor .Inch .he .pcc.fic.1 

action should he carried out). 


attempts to work together to produce a single trans- 
1 at ion. This confirms a hypothesis described in Jo], 
that negotiating requirements conflicts is more effec- 
tive if we start with a precise description of each per- 
son's individual viewpoint. Note that our final version 
was different from all four of the individual versions, 
implying that if the final version is correct, all four 
individual attempts were wrong! 

This leads to the second result, which is that trans- 
lation of informal requirements into a formal notation 
is error prone. All four of our subjects had some ex- 
perience of using such tables, so the problem lies not 
in the correct use of the notation, but in the interpre- 
tation of the informal statement of requirements. The 
requirement we used in the experiment is perhaps an 
extreme example, given its rather convoluted English. 
However, there is enough scope for misinterpretation 
in the process of formalizing the requirements to cause 
us to worry about the fidelity of our formal models. 

The third result is that the whole process was re- 
markably good at identifying ambiguities in the orig- 
inal specification. By producing different interpreta- 
tions and comparing them, we were able to identity a 
systematic pattern of ambiguities in the way the En- 
glish language requirements were written. Hence, even 
if the IV&V team fail to persuade the development 
team to adopt a tabular notation, they can at least 
help them to correct the ambiguities in the English. 

In fact, the development contractors have used the 
tabular notation occasionally, in the most recent ver- 
sions of the specifications. Initially, they resisted the 
IV&V team's requests to adopt a tabular notation, 
largely because of schedule constraints. They have 
now begun to use the notation for revisions of the spec- 
ifications. especially in areas where reviewers had had 
problems with readability. Wc regard tliis as a small 
but important process improvement, inspired by the 
IV&V team. 


4.2 Experiment 2: Analysis of Partial 

Specifications 

Our second goal was to chock some of the properties 
of the FDIR specification that could not he checked by 
hand. One of the important validity checks for these 
requirements is that an act ion is specified for each pos- 
sible combination of failure conditions. Another check 
is that no combination of conditions has conflicting 
actions specified for it. Wc refer to these as coverage 
and disjointness checks, respectively [14]. 

In practice, there were two approaches that IV&V 
could take to verify such properties. They could ob- 
tain the development team's failure model, validate 
this model, and then verify the requirements against 
the model. Or they could generate their own behav- 
ioral model of the requirements as described, verity 
that it is internally consistent, and then validate this 
against their understanding of the system. The lat- 
ter approach was chosen, partly because the IV&V 
team has had difficulty obtaining the original models 
on which the specification is liascd. and partly because 
the latter approach was more likely to overcome anal- 
ysis bias. 

Wc chose SCR. as an appropriate model to perform 
these analyses for a number of reasons. First, the tal>- 
ular notation used in SCR maps onto the AND/OR 
tables wc had already generated in a fairly systematic 
way. Each AND/OR table represents a single row in 
a mode transition table in SCR- Second, there was a 
tool (SCR*) available for checking SCR specifications 
wliich included both coverage and <lisjointncss tests, 
and which had a simulator built in for animating the 
complete state-based model. A model checker was be- 
ing added. Furthermore, the consistency checker in 
the SCR* tool provides counter-examples whenever an 
inconsistency is found. Our early experiments with a 
theorem prover (PVS [14]) were abandoned because 
when a proof failed, it took too long to discover the 
problem. The provision of counter-examples is impor- 








tant in tracing problems back to the informal specifi- 
cation. and in convincing the development team that 
there really is a problem. 

The first step was to produce an SCR model of the 
specified FDIR behavior. At tliis stage we had six 
AND/OR tables, similar to the one shown in Table 
1. representing the six paragraphs, a to f. of section 
2 16.3 of the requirements. Each paragraph isolates 
one failure mode, and specifies an appropriate action. 
We merged these into a single table, modeling cadi 
failure mode as a separate SCR mode (Table 2). 

Merging the AND/OR tables to produce Table 2 
was not straightforward. Although there were a num- 
ber of conditions common to several of the tables, the 
wording varied, and it was not always obvious whether 
similar sounding phrases actually referred to the same 
condition, due to inconsistencies in the use of termi- 
nology. For example the condition '‘the bus has been 
switched in the major (10-second) frame” appeared in 
one paragraph, and "the bus has been switdied in the 
last major frame” appeared in another. We initially 
assumed these to be identical. However, tliis led to an 
inconsistency in the table. In fact the former refers to 
the current frame, wliilc the latter refers to the pre- 
vious frame. There were numerous places where we 
had to make assumptions to proceed, and we carefully 
recorded these as annotations to the original text, to 
be checked with the developers. 

The modes we have identified are not present ex- 
plicitly in the informal specification. Our modes cor- 
respond intuitively to failure modes, but might not 
be a particularly good choice for simulation or model 
checking purposes, because they really express output 
events rather than states. However, they suit our pur- 
pose. as the table in this form can be checked directly 
for coverage and disjointness without completing the 
model. In fact, the complete model would be com- 
plicated: a clock would be needed to implement the 
bus processing frames, together with several timers to 
keep track of historical state. Even then. SCR cannot 
(currently) represent timing conditions on the required 
functions. 

Having created the table, we then checked it for 
coverage and disjointness. Not surprisingly, the ta- 
ble is not disjoint: in fact there is an overlap between 
every possible pair of rows. Analysis of the counter- 
examples provided by the SCR* tool indicates a sys- 
tematic under-specification of the comiitions. The 
original model of the FDIR system was a procedural 
model with an explicit order on the checks that need 
to be performed. The specification does not have this 
explicit ordering, <and the described conditions do not 
adequately express this ordering. However, tliis result 
was not a surprise: the IV&V team had already sul>- 
mitted a report suggesting that the ordering be made 
explicit in the specification. 

Wliilc we were producing this analysis, a new draft 
of the specification was released. The section spec- 
ifying Dus FDIR requirements had been re-written, 
partly due to issues raised by the IV&V team, both 
before and after our first experiment. The new ver- 
sion is much clearer (but docs not use our tables). It 
is also much simpler: several failure modes and at least 


half the conditions expressed in Table 2 have been re- 
moved. and the disjointness problem described al>ove 
has been corrected. 

Hence our formal analysis was redundant before it 
was complete. In practice, it would have been possible 
to perform the analysis much earlier: we delayed the 
work until a full release of the SCR* tool was avail- 
able. However, we can now apply the same technique 
to other parts of the specifications, and expect that 
in some cases it will identify new problems, wliilc in 
others it will supply concrete evidence of known prol>- 
lems. Once the requirements are stable, we plan to 
build a complete model of the FDIR subsystem, and 
use a model checker to study its behavior under rc- 
peated and intermittent fault conditions. 

5 Discussion 

We have described our on-going work with formal 
methods as a tool for an Independent V&V team to 
perform analysis of software requirements. Our ini- 
tial results are encouraging: the translation process 
was extremely valuable in identifying ambiguities and 
improving our understanding of the specification. In 
tliis process, a number of errors were found. Analysis 
of a partial formal specification demon st rat ed an im- 
portant error in the specification, and appears to be a 
powerful means of gaining maximal results from min- 
imal effort. We const, meted just enough of a model to 
test the properties we were interested in. without any 
further commitment to the method. 

However, our experiments have revealed two relat ed 
problems: it is hard to guarantee fidelity between in- 
formfil and formal specifications, and it is hard to 
manage consistency between partial specifications ex- 
pressed in different notations. 

Although the major finding of our formal analysis 
is valid, we arc not confident that the partial model 
is faitliful to the version of the developer’s specifica- 
tion on which it is based. This fidelity issue is more 
of a problem in IV&V than in development. A for- 
mal model developed by the IV&V team cannot re- 
place the informal specification. The IV&V team must 
therefore either persuade the developers to adopt for- 
mal notations themselves, or take care to maintain fi- 
delity between the developers' informal specifications 
and their own formal models. With the current state 
of practice, wholesale adoption of formal methods \iy 
the developers on an existing project is unlikely (4]. 

The fidelity problem is important to IV&V because 
the formal models developed by IV&V are produced 
for the purposes of checking the developer's specifi- 
cations. The models arc only useful for this purpose 
if they arc accurate representations of the developer s 
specifications. Also, when analysis of the formal inod- 
ds reveals problems in the specifications, these prob- 
lems must be traced back to the informal specification 
before they can be reported. 

Although the fidelity problem seriously affects the 
utility of any formal analysis performed by the IV&V 
team, we should point out that it docs not affect all 
the benefits of formal specification. The process of 
translating pieces of the informal specification into a 
formal notation has benefit not just for the analysis 
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Tnb\c 2* An SCR Mode transition table. Each of the central columns represents a condition, showing whether it 
should i>0 true or false; means Mont care**; k @T indicates a trigger condition for the mode transition. Tdic 
four columns of table 1 correspond to the last four rows of this table. The semantics of SCR require tins table 
to represent a function, so that the disjunction of all the rows covers all possible conditions (coverage), and the 
conjunction of any two rows is false (disjoint ness). 


that it leads to. but also for the removal of ambiguities 
and for improved understanding. For this benefit, it 
is the process of formalization, rather than the end 
product that is important. 

The fidelity problem is really a special case of a 
more general problem* management of consistency be- 
tween partial specifications expressed in different no- 
tations. For instance, the AND/OR tables have a 
clear relationship with the SCR mode tables, but if 
we make a correction to one of the AND/OR tables, 
it is fairly tc<lious to identify the corresponding cor- 
rection in the SCR tables. Similarly, eacii time the 
developers issue a new informal specification, we need 
to update our tabular representations. Although it 
may seem that the use of both AND/OR tables and 
SCR models together would compound tliis problem, 
the opposite is true. The AND/OR tables mapped 
clearly onto the textual requirements, while the rela- 
tionship between the AND/OR tables and the SCR. 
model was relatively straight forward. Therefore, the 
use of AND/OR tables as an intermediate representa- 
tion reduced the traceability gap. and made it easier 
to keep the formal model up to date. There remains, 
however, a significant bookkeeping problem. 

There is a growing body of work on handling incon- 
sistency in specifications. Our previous work demon- 
strated how to delay the resolution of inconsistency, 
and provided a generic framework for expressing con- 
sistency relationships [6]. Other work has taken con- 
sistency checking further, making use of semantic 
models underlying a method to determine what con- 
sistency rules arc needed and how to operationalize 
them. For example. Hcitmcycr's work with consis- 
tency checking in SCR [9] uses the semantics of SCR 


to define a series of consistency rules ranging from sim- 
ple syntactic checks (c.g. that all names arc unique) 
to sopliisticatcd properties of tables (c.g. coverage 
and disjointness). Similarly. Leveson’s work on con- 
sistency checking in RSML [8] uses the semantics of 
the statcchart formalism to determine a set of consis- 
tency rules that can he tested, tractably, using a high 
level al>stract model. In both these approaches, the 
completeness of the formal specifications is important, 
and consistency checking is seen as part of the process 
of obtaining a complete, consistent specification. 

Unfortunately, these approaches do not help with 
consistency checking between partial specifications ex- 
pressed in different notations. Because the IV&V pro- 
cess is concurrent with and complementary to the de- 
velopment process, there is an unusually large amount 
of flexibility in how a formal method can be used. 
There is no need to make a commitment to any one 
formal notation, just as there is no need to develop 
complete specifications. In fact, the aim of the IV&V 
agent is not to perform complete analyses, hut to do 
just enough analysis to cheek specific aspects of the 
software. Development of complete formal models is 
therefore unnecessary and may be counter-productive. 
For example, in our second experiment, the limited 
analysis we performed on a partial model was suffi- 
cient to reveal a major problem; the existence of this 
problem meant, that any further effort to complete the 
model would have been wasted. 

While the use of partial specifications offers greater 
flexibility in the use of methods and tools, it also 
means that we do not have a well-defined method from 
which to generate a set of consistency relationships. 
There arc implicit consistency relationships between 
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the assorted partial specifications drawn from differ- 
ent methods, hut there is no overall ‘method* to to 
tell us what these relationships are. Actually, there 
is a method: the problem is that it is implicit, and 
to some extent is generated on the fly. For example, 
there is a method for generating SCR mode tables 
from the AND/OR tables, but the method was not 
defined before we did it. With some effort, we could 
formalize this method, and define semantic relation- 
ships between the two types of table. However, this 
effort will only be worthwhile if we intend to re-use 
the method extensively. In the meantime, we would 
like to have tools to help us keep track of consistency 
relationships in our opportunistic use of partial spec- 
ifications. 

In our previous work defining consistency relation- 
ships between viewpoints, we assumed that the ma- 
jority of such rules are definal by the method [6]. The 
viewpoints framework explicitly supports the process 
of method definition, in which, among other things, 
the inter- viewpoint relationships arc defined. Hence 
the general problem of defining arbitrary relationships 
between any two notations is avoided. However, we 
also recognized that some consistency relationships 
could not be defined in this way. and gave the ex- 
ample of a user-defined synonym relationship between 
two different labels. We also outlinal an approach to 
discovering such relationships through low level pro- 
cess monitoring. We now regard tliis type of consis- 
tency relationship as vital to any approach involving 
partial specifications. 

Without a method to define a priori consistency re- 
lationships. we arc forced to discover the relationships 
as the work proceeds. In fact this is not as hard as 
it sounds. By recording low level act ions on the par- 
tial specifications, we begin to build up a fine-grained 
process model, which can provide information about 
consistency relationships. For example, by observing 
cut and paste operations during the creation of our 
AND/OR tables and our SCR mode tables, it is pos- 
sible to determine the relationship between rows in the 
AND/OR tables and rows in the mode table. In the 
weakest case, tliis will provide us with a simple trace- 
ability link. In fact, we believe we can do better than 
tliis. There is enough information in the edit actions 
not just to identify traceability links, but to define 
the relationship expressed by the link. For example, it 
should he possible to determine enough information to 
define a consistency rule that can automatically check 
that cadi column of the AND/OR table is consistent 
with its corresponding row in the mode table. We 
plan to explore this avenue further, by capturing and 
analyzing this kind of process information. 

6 Conclusions 

Tliis paper has described our initial work in the use 
of formal methods in an IV&V project. We have dis- 
cussed how the demands placed on methods and tools 
in IV&V arc different from their use in a development 
context. We have also discussed how IV&V can act 
as a process improvement agent, and hence can be a 
fruitful way of introducing formal methods into large 
projects. 


As with all potential uses of a new method, any 
extra effort needed to use the method must he more 
than offset by the benefits it brings. Use of a method 
in IV&V is no different. We can divide the benefits of 
using a formal method such as SCR into two areas: 

1. The process of translating portions of a speci- 
fication into a tabular notation helps to detect 
ambiguities and increase readability, even if the 
translation is only partial. The process can also 
he used to catch misunderstandings, thus increas- 
ing the confidence that the IV&V team is inter- 
preting the specification correctly. The process 
of having several analysts produce their own tab- 
ular translations was particularly useful in tliis 
respect. Differences in the tallies they produced 
allowed us to pinpoint exactly what the disagree^ 
merit was about. 

2. The resulting tables can be analyzed for at- 
tributes such as coverage and disjointness. This 
is a substantial contribution to the IV&V team s 
efforts to check the technical integrity of the spec- 
ifications. Such iittributes arc p<yticuiarly hard 
to analyze from the informal specifications. Most 
importantly, this analysis can be conducted with- 
out the need to build complete models. 

The problems we encountered in applying formal 
methods were as follows: 

1. The process of translating into a formal notation 
is error-prone. Only by duplicating the transla- 
tion effort were we aide to discover just how much 
scope there is for misinterpretation. Luckily, the 
resulting tables arc very readable. Therefore it 
is much easier to compare different tables than 
it is to compare different versions of the informal 
specification. 

2. For IV&V, fidelity and traceability between the 
informal and formal specifications is difficult to 
guarantee. The value of any analysis carried out 
by IV&V on the formal model is entirely depen- 
dent on how faitliful the formal model is to the 
developers informal specification. The rV&V s 
formal model can not he used in place of the in- 
formal specifications produced by the developers. 

3. Opportunistic use of partial specifications means 
that there is not a wcll-dcfinal method from 
which to derive consistency rules. Maintenance of 
consistency in our partial specifications became a 
real problem. 

The problems of consistency checking in partial 
specifications written in different notations is impor- 
tant enough to warrant more attention. We plan to 
study the problem in more detail by developing a set 
of tools based on the ViewPoint framework [7], which 
will allow us to model relationships between partial 
specifications written by different people. We arc also 
exploring bow this problem relates to that of linking 
test ease scenarios to requirements [2]. Finally, we arc 
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continuing the experiments described in this paper hy 
examining how model checking can be used to validate 
the specifications. 

Acknowledgments 

Our thanks arc due. to Chuck Ncppach and Dan 
A feCaugherty for many interesting discussions oj the 
work presented here . and to Frank Schneider. Ed- 
ward Addy . John Hinkle . George Sabolish . Tofld Mont- 
gomery and Dutch Neal Jor detailed comments on ear- 
lier drafts of this paper . This work is supported by 
NASA Cooperative Research Agreement NCCW-OOJO 

References 

[1] V. Basili. The experience factory and its rclation- 
sliip to other improvement paradigms. In Proceed- 
ings of the 4 th European Software Engineering 
Conference . . Garmish-P arte nkirr hen. Germany . 
September 1993. 

[2] J. Callahan and T. Montgomery. An approach 
to verification and validation of a reliable mul- 
ticast protocol. In Proceedings of the ACM In- 
ternational Symposium on Software Testing and 
Analysis (ISSTA). January 1996. 

[3] D. Craigen. S. L. Gerhart, and T. Ralston. 

Formal methods reality check: Industrial us- 

age. IEEE Transactions on Softxrarc Engineering. 
21(2):90-98. 1995. 

[4] D. H. Craigen. S. L. Gerhart, and T. .1. Ral- 

ston. An international survey of industrial ap- 
plications of formal methods, vol 1: Purpose, 

approach, analysis and conclusions. Tcclinical 
Report NRL/FR/554^ 93-9581. Naval Research 
Laboratory. 1993. 

[5] S. M. Easterhrook. Handling conflict between do- 
main descriptions with computer supported nego- 
tiation. Knowledge Acquisition: An International 
Journal. 3(4):255-289. 1991. 

[6] S.M. Easterhrook and B. A. Nuseibch. Managing 
inconsistencies in evolving specifications. In Sec- 
ond IEEE International Symposium on Require- 
ments Engineering, pages 48—55. March 1995. 

[7] S. M. Easterhrook and B. A. Nuscibch. Us- 
ing viewpoints for inconsistency management. 
DCS/IEE Software Engineering Journal 11(1). 
January 1996. 

[8] M. Hcimdahl and N. Leveson. Completeness and 
consistency analysis of state-based requirements. 
In Proceedings of the 17 th International Confer- 
ence on Software Engineering . pages 3-14. April 
1995. 

[9] C. Hcitcmcycr. B. Labaw, and D. Kiskis. Con- 
sistency checking of scr-stylc requirements speci- 
fications. In SccoTul IEEE International Sympo- 
sium on Requirements Engineering . pages 56^63. 
March 1995. 


[10] C. Heitmeyer. A. Bull. C. Gas arch, and B. Labaw. 
Scr*: A toolset for specifying and analyzing re- 
quirements. In Tenth Annual Conference on 
Computer Assurance (COMPASS 95). pages 
109-122. June 1995. 

[11] D. Jackson and C. A. Damon. Elements of style: 
Analysing a software design feature with a coun- 
terexample detector. In International Symposium 
on Software Testing and Analysis (ISSTA). pages 
239-249. January 1996. 

[12] Jet Propulsion Lab. Cost-effectiveness of software 
independent verification and validation. Techni- 
cal Report NASA RTOP 323-51-72. NASA JPL. 
October 1985. 

[13] R. O. Lewis. Independent Verification and Valida- 
tion: A Lifecycle Engineering Process for Quality 
Software. J. Wiley & Sons. 1992. 

[14] S. Owre. J. Rushby. and N. Shankar. Analysing 
tabular and state-transition specifications in pvs. 
Tcclinical Report CSL-95-12. Computer Science 
Laboratory. SRI International. 1995. 

[15] H. Saiedain. J. P. Bowen. IL W. Butler. D. L. Dill. 
R. L. Glass. A. Hall. M. G. Hinchcy. C. M. Hol- 
loway. D. Jackson. C. B. Jones. M. J. Lutz. D. L. 
Parnas. J. Rushby. J. Wing, and P. Zavc. An 
invitation to formal methods. IEEE Computer. 
29(4):16-30. 1996. 


9 



